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METHOD AND APPARATUS FOR AUTOMATIC ELECTRONIC 
REPLACEMENT OF BILLBOARDS IN A VIDEO IMAGE 

The present invention relates to a method and apparatus for 
5 automatically replacing billboards in a video image. 

The present invention has particular use in electronic replacement 
of billboards in a stadium or other venue but can be used to provide 
accurate data relating to camera orientation for other purposes. 

10 

In previous systems it has been proposed to electronically replace 
billboards in a stadium which are viewed by a viewer on television* The 
billboards in the stadium are televised by a TV camera and the boards are 
electronically altered so that the TV viewer at home sees a different board 
15 to the spectator in the stadium or other venue. 

The known systems such as those described in US 5,266,933, an 
apparatus and method for electronically altering video images is disclosed. 
The apparatus and method disclosed in the US patent and also in US 
20 patent 5,353,392 whilst theoretically allowing replacement of billboards 
do not solve the many practical problems encountered in real 
environments. Most of these problems are related to the recognition and 
replacement processes. 

25 Relying entirely on pattern recognition techniques which utilize only 

the video signal to identify and localise billboards for replacement 
introduces major problems which affect the practical value of such a 
system. 

30 Clearly, any pattern recognition scheme, including those described 

1 
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in US 5,264,933 and US 5,353,392 must rely on useful visible features in 
the image that can be compared with pre-defined descriptions. Such 
features should be located inside the billboard or at its neighbourhood. 

5 In realistic situations, the visibility of these features might change, 

continuously or otherwise from practically zero to a some threshold 
visibility which allows the pattern recognition scheme to work properly. 
These changes can occur in the direction of growing or reducing visibility. 

10 Such situations include : 

• Acceleration or de-acceleration of camera motion 
introducing a huge amount of blur. 

• Excessive zooming-in or zooming-out of the billboard. 

• Excessive occlusion by players. 

15 • Entering or exiting a camera's field of view by any 

combination of pan, tilt and zoom operations. 

• Any combination of the above mentioned mechanisms. 

Therefore, in practical situations, a continuous replacement of 
20 billboards, is not possible. Even if an interrupted replacement was 
allowed, it would require a delay of at least a few seconds to decide 
whether the resulting replacement interval is acceptable or not. Such a 
delay is usually not permitted in live broadcasting of sports events. 

25 Replacing arbitrary billboards introduces further problems. A 

seamless replacement requires to identify the foreground objects occluding 
the billboard in order to inhibit replacement at places of occlusion . 
Foreground objects mainly consist of players but also the ball or other 
objects. Consider now a player with a red shirt, occluding a part of a 

30 similarly red portion of a billboard. Colour contrast cannot be used 
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robustly to identify occlusion. Furthermore, since the player is a non- 
rigid object, motion or shape information cannot be used accurately 
enough to guarantee perfect replacement. 

5 Another problem which may arise in practical situation is resolution 

of billboard identity. Consider two identical billboards positioned at two 
different locations in the arena. Suppose different replacement billboards 
are assigned to each of these physical billboards, then one must be able to 
tell which one is which. This can prove to be extremely difficult 
10 especially if no unambiguous features are visible. 

This invention describes a robust system for billboard replacement, 
based on some or all of the following key-elements: 

15 • Physical billboards which are coloured properly to enable 

the efficient detection of occlusion by chroma-key 
techniques. 

• Colour variation or a pattern within the physical billboard 
for further enhancing the performance of the image 

20 processing methods . 

• Pan, tilt, zoom and focus sensors attached to the camera, 
which enable after a proper set up procedure to estimate the 
presence and location of billboards in any given video field. 

• Image processing methods and their embodiment which 
25 enable to refine the sensors' estimates. 

The present invention therefore has for a first object to provide a 
physical billboard which comprises a chroma-key surface and also 
provides means for enabling an electronic replacement billboard to be 
30 substituted for the chroma-key board. 

3 
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The present invention therefore provides apparatus for automatic 
replacement of a billboard in a video image including a video camera for 
viewing the billboard, characterised in that the billboard comprises a 
chroma key surface and that the apparatus further includes a chroma-key 
5 unit operative to detect at least one chroma-key colour and in which the 
chroma-key colour is adjustable to conform to the colour of the chroma- 
key surface. 

The present invention also provides a method and apparatus which 
10 enables identification of the location of a billboard or other static object 
in a stadium or other venue in any weather conditions with any panning 
speed of the camera and with any other change in camera parameters. 

The present invention therefore provides apparatus for automatic 
15 electronic replacement of a billboard in a video image including an 
automatic camera orientation measurement apparatus including motion 
measurement means operative to measure the Field of View (FOV) of the 
TV camera relative to a known reference position. 

20 The present invention also preferably provides apparatus for 

automatic electronic replacement of a billboard in a video image, including 
image processing means for processing video signals generated by the TV 
camera, in which said processing means includes calibration means for 
periodically automatically calibrating the motion measurement means, 

25 apparatus in which the motion measurement means includes means for 
measuring the pan tilt, zoom or focus of the camera relative to known 
reference positions and apparatus in which the motion measurement means 
includes means for measuring the pan tilt, zoom and focus of the camera 
relative to known reference positions. 

30 
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The present invention therefore uses dynamic recalibration to 
correct for residual sensor errors or abberations in an imperfect model and 
for sensor drift over time. Thus it is possible in accordance with the 
present invention to use less stable sensors and the apparatus and method 
5 in accordance with the present invention can accommodate movement in 
the camera position. The image correction process for calibration of the 
sensors eliminates the necessity to keep the sensors stable by mechanical 
means by recalibration automatically with reference to the video image. 

10 In the initial set up procedure corrections can be incorporated for 

calibration for billboards which are, for example, not in the centre of the 
Field of View - for example a billboard which is in the top left-hand 
comer of the screen can be adjusted by, for example, 3 pixels to take into 
account abberations in the camera, 

15 

Problems which arise in the prior art systems are firstly when the 
billboard is either substantially totally occluded or secondly, is occluded 
by an object, such as a player, of the same colour as the real sign on the 
billboard. 

20 

This can firstly, as explained above, lead to non-recognition of the 
billboard and also secondly, to difficulty in satisfactory replacement of 
the billboard. 

25 In the first case the real billboard can have already been replaced 

in the video image but if the camera zooms into a close up or if a different 
camera is used for the close up then the lock may be lost due to only a 
very small portion of the billboard being in view. In the second case the 
player may have on a strip which is the same colour as the billboard. The 

30 prior an systems propose to distinguish the billboard from the player on 
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the basis of movement if the colours are the same and to analyse the 
"moving" pixels to determine occlusion. This is reasonable in theory but 
fails in practice since not all players are moving at all times. Thus, if a 
number of players move in front of a billboard and one player remains 
5 after the others have moved on, the electronics will not be able to 
distinguish on movement grounds. Since colours are distorted by 
floodlighting, shadows, differences in reflectivity and different lighting 
conditions for a foreground player and a background billboard, there will 
be occasions in practice where the system fails. In such cases either the 
10 original billboard may re-appear on the video image or the replacement 
billboard will not be accurately occluded. 

It is again possible to introduce a delay in the video transmission 
to enable the electronic signal processing to be more accurate but this does 
15 not solve the practical problem where a plurality of players move in 
different directions to occlude a billboard. The necessary delay is 
considered unacceptable and will in any case not solve all of the above 
problems. 

20 In accordance with a preferred embodiment of the present invention 

it is proposed to replace the real billboards with chroma-key panels or 
with delineated areas forming chroma-key panels. 

Chroma-key is essentially an occlusion technique allowing, for 
25 example, a news reader to stand and move about in front of a chroma-key 
board, usually coloured blue or another suitable colour. The news reader 
(foreground) is distinguished from the chroma-key board (background) by 
colour differentiation and can thus move in front of the replacement 
background with normal occlusion of the foreground and background. 
30 This technique is very well known in television studio systems and is 
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described fa numerous US patents, including US 2,974,190 and 
4,200,890. 



< A > 



In the present invention the billboards are replaced by chroma-key 
panels or with delineated areas forming chroma-key panels. 

In the prior art however only one chroma-key area is identified and 
this area is accurately lit. In the present invention several billboards may 
be situated at various locations within a stadium and the present invention 
enables electronic billboards to be substituted for the several billboards. 

Also in a virtual set studio application, the camera is typically 2-10 
metres away from the foreground and the entire field of view is usually 
replaced. In comparison, a billboard may be several hundred metres from 
the camera and therefore a replacement system using sensors is much 
more susceptible to sensor errors: 



Due to the large focal distances, the same sensor accuracy 
will translate to larger geometric registration errors. 
Consider a rotary encoder of 81000 pluses/revolution, then 
the angular precision is 0.0044 degrees or 75 micro-radians. 
The repeatability is twice as bad. Consider a shooting range 
of 100 m with a field of view of 4 metres, then the FOV is 
40 milli-radian. The error translates to 768 * 150 / 20000 
= 2.88 pixels. 

Since the field of view includes many stationary objects 
(including billboards) which are not. replaced, the human 
observer will be much more sensitive to the registration 
errors. Additional errors may originate from lens distortion,- 
rotation axis which does not pass through the focal point, 
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< A> 

US Patent No. 5.353,392 discloses an electronic replacement 
billboard system in which a real billboard is replaced in a video image by 
a virtual or electronic billboard. Occluding obstacles placed between the 
video camera and the billboard are identified by lack of correlation with 
the colour or luminance of the_billboard. 

PCT WO94/05118 discloses a virtual studio system using a 
background chroma-key panel. The panel comprises a uniform pattern of 
two chroma-key colours with differing hue and/or brightness which 
enables the position of a camera relative to the background chroma-key 
panel providing that the initial camera position is known. 
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non-zero roll angle, etc. 

Chroma-key is basically a technique for studios where the 
illumination is carefully designed and controlled and the controls of the 
5 chroma-keyer are carefully adjusted for the specific arrangement of blue- 
screen colour and illumination. 

In a sports event, the conditions may be highly non-ideal and 
require some modification to the chroma-key algorithms. In particular, 
10 the keyer parameters should be adapted to the specific billboards being 
replaced due to changes in illumination across the arena. 

Accordingly in the present invention it is proposed to use chroma- 
key panels and to replace these in the video image by the replacement 
15 billboards. 

Since it is necessary for perfect occlusion that the players or other 
occluding objects are of different colour to the chroma-key panels, it is 
proposed in a further preferred embodiment to provide chroma-key panels 
20 in which the colour of the panel may be changed, for example, by using 
a rotating billboard structure which is known in the art. One side could, 
for example, be blue and another green. Green may be preferable in a 
sports environment since players tend not to wear green as this would not 
contrast with the background sports surface. 

25 

In a further preferred embodiment and in particular where a 
plurality of billboards require replacement, a patterned chroma-key board 
is used. The pattern may be of any suitable shape but is preferably 
selected to be suitable for the size and shape of the billboard or series of 
30 billboards and also to the anticipated video conditions. Thus if a billboard 

8 
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is only able to be viewed from a long distance then a different pattern will 
be selected to a billboard which is to be viewed in close up. 

The pattern may comprise different colours or may be different 
5 shades of the same colour. The pattern may comprise vertical and 
horizontal lines or may comprise a decorative pattern, a discernible 
advertisement, company logo or other suitable wording which may be 
more aesthetically acceptable. 

10 The use of a pattern allows further discrimination of the position of 

the camera and may allow movement of the camera from a fixed position. 

The camera orientation data can be transmitted together with the 
video signal and will identify the position of the billboard in any weather, 
15 lighting or occlusion conditions. No reference is required to any feature 
within the sports venue to identify the position of the billboard. 

The camera sensors can be accurate to a few pixels or in physical 
terms to approximately 1 cm at a range of about 100 metres thereby 
20 enabling accurate replacement of any billboard. The re calibration can be 
carried out continuously or only periodically, particularly if an initial 
adjustment of the calibration of billboards not in the centre of the FOV is 
recorded on set up. 

25 By use of the chroma-key techniques there is no requirement to 

transmit any occlusion data since this can be readily inserted at a receiver 
and the occlusion inserted in the normal manner. 

However when situated in different locations in the stadium, the 
30 boards, including those areas of chroma-key colour which are delineated 

9 



CA 02231376 1998-03-06 
WO 97/09822 PCT/GB96/02224 



as chroma-key boards, hereinafter referred to as billboards, will be subject 
to varying and differing lighting conditions. 



The problems which the inventors have realised is that a chroma- 
5 keyer will be set to detect a particular chroma-key colour and under 
different lighting conditions the colour of a billboard (as defined to include 
areas where billboards are to be inserted) will change. Thus the apparatus 
will be unable to accurately identify the various billboards situated around 
the stadium. 

10 

The present invention also therefore provides apparatus comprising 
a billboard map memory for recording the position of plurality of 
billboards and for recording the chroma-key colour of each billboard to be 
replaced to thereby provide a chroma-key colour map for a stadium. 

15 

A further problem which the inventors have realised is that if a 
plurality of different chroma-key colours are transmitted to a remote 
receiver to provide substituted billboards at a local distribution centre, 
then the chroma-key equipment at such a centre must also be equipped 
20 with selective chroma-key mapping. This substantially complicates the 
local receiving apparatus and it is an object of the present invention to 
overcome this problem. 

The present invention therefore provides apparatus further including 
25 means for transmitting, for each billboard, to be replaced a perfect 
background colour key signal. 



The present invention further provides apparatus further including 
means for transmitting the co-ordinates of each billboard to be substituted 
30 and apparatus further including receiving apparatus for reception of the 

10 
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billboard coordinates and the perfect background chroma-key colour and 
including means for providing a combined video picture incorporating 
occlusion in the coordinate areas identified by the perfect background 
chroma-key colour. 



In a preferred arrangement within a stadium or other sports venue 
real billboards with normal advertising material will be situated on one 
side of the stadium to be viewed by a first plurality of cameras and 
chroma-key billboards will be situated on another or the opposite side to 
10 be viewed by a second plurality of cameras. This, for example, the home 
nation may view the normal billboards, with the international TV audience 
seeing only substituted boards. 

The present invention also provides a method for electronically 
15 replacing a billboard in a video image display, generated by a camera, 
comprising the steps of : 



5 



20 



a. 



identification of the position of a rectangular chroma-key 
colour billboard in a stadium or other venue, said 
identification step comprising specifying on the video display 
the chroma-key billboard to be replaced by identification of 
its four corners at a first camera position; 



b. 



c. 



storing the identification information; 

monitoring the movement of the camera in pan, tilt and 



zoom; 



25 



d. 



storing the monitored movement of the camera on a field by 
field basis; and 



30 



e. 



analysing the size and position of the chroma-key billboard 
to be replaced from the information recorded in its first 
known position and the stored movements of the camera to 
provide information relating to the size, perspective and 
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position of the chroma-key billboard in the present video 
field; 

storing in a billboard replacement store a replacement 
billboard to be used in replacement of the chroma-key 
billboard in the stadium; 

electronically altering the size and perspective of the 
replacement billboard in accordance with the camera motion 
information to conform to the size and perspective of the 
chroma-key billboard to be replaced in the present video 
frame; and 

electronically replacing the chroma-key billboard in the 
present video frame by the replacement billboard. 

In a preferred embodiment the step of analysing the size and 
15 position of the chroma-key billboard to be replaced comprises a further 
step of analysing a plurality of video scan lines to provide fine adjustment 
information relating to the exact size, perspective and position of the 
billboard to be replaced. 

20 In a further preferred embodiment the chroma-key billboard to be 

replaced may be of a blue or green shade, due to the fact that these 
colours are rarely found in human skin and hair. 

In a still further preferred embodiment the chroma-key billboard is 
25 patterned with a pattern of a suitable shape for the purpose of facilitating 
the fine adjustment process mentioned above. The step of analysis of the 
size and position of the billboard comprises the analysis of the pattern to 
ascertain the exact position of the billboard. 



WO 97/09822 



f. 

5 



10 

h. 



30 



In a further embodiment the correction of the sensor-based 
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prediction by the analysis of the pattern will be controlled by a figure of 
merit (accuracy estimate) for the analysis, which will be computed 
automatically. 

5 In a further embodiment the step of electronically replacing the 

billboard in the present video field by the replacement billboard include 
the step of superimposing occluding objects by use of the chroma-key 
techniques. 

10 In a further preferred embodiment the billboard to be replaced can 

be changed to best match the colours and shades of colours on the players 
costumes, for the purpose of providing a good contrast between the 
billboards and the players. For example, if these costumes contain shades 
of blue, then a green billboard may be selected. 

15 

The backing colour can be selected between blue, green and red. 
In order for the chroma-keyer to calculate all parameters necessary to 
perform proper image compositing, the system requires a sample of the 
background colour as reference. This step can be done automatically by 
20 scanning the image and detecting the purest and brightest colour. 
Advanced chroma-keyers enable the user to manually select the area to be 
sampled. 

In a further preferred embodiment the chroma-key apparatus will 
25 have a multiplicity of set up conditions, each corresponding to a different 
region of the stadium. The camera pan, tilt and zoom information will 
allow to load the corresponding set up conditions. 

In a further embodiment, the fine adjustment information will be 
30 used to compensate drift errors of the sensors. In a practical situations, 

13 
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the sensor error will have a significant portion which is at temporal 
frequencies which are much lower than the video field rate. Thus these 
sensor-induced errors can be reliably estimated from good video field and 
subtracted from subsequent measurement. 

5 

The present invention also provides apparatus for carrying out the 
method of electronically replacing the billboard as specified hereinbefore. 

Embodiments of the present invention will now be described, by 
10 way of example with reference to the accompanying drawings in which :- 

Figure 1 shows a stadium or other venue illustrating the apparatus 
according to the present invention; 

Figure 2 shows the video image of the stadium as seen by the 
15 camera in a first position; 

Figure 3 illustrates a stadium with billboards in several different 
positions; 

Figure 4 illustrates a zoomed camera shot of a billboard illustrating 
the problem with prior art systems; 
20 Figure 5 shows a patterned chroma-key billboard for use with the 

present invention; 

Figure 6 shows in block diagrammatic form circuitry associated 
with the camera arrangement of Figure 1 for transmitting video data and 
camera orientation data; 
25 Figure 7 shows in block diagrammatic form receiver circuitry for 

co-operation with the transmitter circuitry of Figure 6; 

Figure 8 shows a flow diagram for the operation of the circuitry of 
Figure 7; 

Figure 9 shows an arrangement for a billboard setup data store; 
30 Figure 10 shows an arrangement for billboard setup data; 

14 
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Figure 11 shows a flow diagram for perspective transformation 
computation; 

Figure 12 shows an arrangement for camera intrinsic parameters 

store; 

5 Figure 13 shows equations for dynamic recalibration; 

Figure 14 shows a flow diagram for dynamic recalibration; 
Figure 15 shows the process of recalibration; 
Figure 16 illustrates the problems relating to siting of billboards at 
different locations within a stadium with differing lighting conditions; 
10 Figure 17 shows a graph of minimum and maximum levels for U 

& V illustrating the operation of a chroma-keyer; 

Figure 18 shows a billboard with occluding object illustrating the 
principle of adjustment of the chroma-key colour for a billboard; 

Figure 19 shows an exemplary remote receiver for reception of 
15 billboard coordinate data and perfect chroma-key colour, occlusion being 
effected by chroma-key techniques; 

Figure 20 shows an alternative arrangement for the billboard set up 
data store illustrating an alternative embodiment; 

Figure 21 shows a flow diagram for dynamic set up procedure for 
20 camera panning for use with the billboard set up data store of Figure 20; 
and 

Figure 22 shows a flow diagram for dynamic set up procedure for 
camera tilting for use with the billboard set up data store of Figure 20. 
With reference now to Figures 1 to 4 the principle of the present 
25 invention is now explained. 

In a stadium or other venue 10 billboards 14,16,18 are installed at 
the side of a pitch represented by markings 12. These billboards are 
viewable by a camera 20. Billboards 15,17,19 may be present on the 
30 opposite side of the stadium for viewing by a further camera 21. The 

15 
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stadium terraces/seating are shown diagrammatically by the lines 1 1 . 

Camera 21 may in a preferred example be a normal TV video 
camera and will transmit its output video signal directly to a first feed 
5 which may serve the local population. Although we refer to camera 20 
or 21, it may be clearly understood that there could be a plurality of 
cameras on each side of the stadium providing differing views. 

Camera 21 in a preferred embodiment will televise boards 15,17, 19 
10 which will be transmitted to the local population in an unchanged mariner. 

Camera 20 will in this preferred embodiment transmit a feed to an 
international audience. Camera 20 is equipped with orientation sensing 
means which preferably comprises one or more of the following : 
15 pan measurement means 24; 

tilt measurement means 25; 

zoom measurement means 26; and 

focus measurement means 28. 

20 Suitable sensors may comprise the Virtual Reality Encoder from 

RADAMEC EPO, Bridge Road, Chertsey, Surrey KT16, 8U, England. 

Dependent on the allowed mobility of the camera only one, several 
or all of these may be required. For example, if camera 20 is fixed in 
25 pan and tilt and focus and can only zoom, as in the case of some remotely 
controlled unmanned cameras then only the zoom parameter need be 
measured. 



30 



Most cameras in sports stadiums can zoom, tilt and pan and it is 
assumed that these parameters are measured for each camera as now 
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explained. The focus is assumed to be fixed but in similar manner the 
parameter could be added if required. 

Figure 2 shows the video image as seen by a viewer and in 
5 particular by the operator of the equipment. The camera 20 is zoomed, 
panned and/or tilted to "centre" the billboard 14 in a suitable position and 
at a reasonable size. With reference to Figure 7 each billboard is then 
viewed at a receiver and its position is marked preferably by using a touch 
screen 700, or keyboard mouse 702 and marking the four corners. The 
10 positions are stored in a store 704. 

For billboards higher in the stadium such as 30 (Figure 3) a 
correction factor for the camera may be stored dependent on the tilt 
position of the camera. 

15 

Each billboard position is stored in store 704 together with the 
camera parameter information at the reference position for the camera 20 
obtained from the camera parameter information which is correct at the 
time that the billboard position is stored. 

20 

The following procedure is preferably repeated for each of the 
cameras and for each of the target billboard : 

1 . Point the camera at the target to obtain a stable unoccluded view of 
25 the target. Adjust the zoom to get a large view of the target yet 

keeping the whole target within the field of view. 

2. While the camera is not moving trigger an acquisition device, to 
grab a picture of the target, as well as the corresponding readings 
of the sensors. 

30 3. Mark the corners of the target, on the video image. 
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Preferably a corner detector is used to pinpoint the corners of the 
target at sub-pixel precision. 

This camera parameter information is obtained (Figure 6) from the 
5 sensors mounted oh the camera and the camera movement is referenced 
to a first or fixed reference position for each parameter. The movements 
of the camera are sensed and the signals are fed into a combiner circuit 24 
and then to a transmit buffer 36 from which the combined video and 
position data signals are transmitted. 

10 

During set up, at the receiver (Figure 7) the receive buffer 706 
receives the signals and feeds these to a splitter 708. The video signal is 
stored and delayed in a suitable store 710 and the camera parameter data 
is extracted and stored in store 712. 

15 

In set up the VDU 700 is used to mark each billboard that may 
require replacement. The camera 20 is panned etc to move each billboard 
into a suitable position on the screen and its position is recorded in the 
billboard store 704 together with the camera parameters obtained from 
20 store 712 via processor 714. 



25 



30 



A replacement billboard store 716 stores a plurality of replacement 
billboards and these are selectable to be able to replace the original 
billboard. 

The replacement billboard is in operation inserted into the video 
signal in a combiner 718 to provide a modified output video signal 720. 

The setting up procedure can also identify billboard locations and 
camera parameters for several cameras by storing a camera ID from a 
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source 30 (figure 6). Thus, billboard position store 704 will store separate 
lists of billboard data for each camera. 

The operation of the system will now be described with reference 
5 to a single billboard and a single camera 20. 

With reference to Figure 4, it is assumed that billboard 14 enters 
the field of view in an enlarged form on the left-hand side of the screen 
as camera 20 pans following zooming from the Figure 3 position. 

10 

The camera orientation data is constantly being received by the 
receiver and the processor 714 will constantly match on a pixel by pixel 
basis the video image with the known billboard position stored in store 
704. As soon as the billboard appears in the video image the pixels 
15 representing the billboard will be identified and the replacement billboard 
pixels which relate to those pixels will be substituted in the combiner 718. 
The delay will be minimal since the identification of the pixels is by an 
address correlation process which will be virtually instantaneous. 

20 After a period of time the camera sensors may drift and in this case 

the replacement billboard may not exactly align with the original. This 
may only be by one or two pixels and may not be discernible to the 
viewer. To correct this two solutions are possible. Firstly, the billboard 
position can be manually restored periodically at a suitable time, for 

25 example, when a camera is not active. This requires the co-operation of 
the operator. 

Secondly, a comparison can be made on a pixel by pixel basis of 
the billboard against an original stored billboard and an adjustment of the 
30 reference camera parameters can be made in billboard position store 704. 

19 
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This process can be done automatically at either set intervals or when the 
processor 714 has a suitable time slot. 

The essential steps of a preferred recalibration process are to 
5 perspectively transform the current video image using the camera data to 
provide an estimated transformed model. A stored image of the billboard 
is then compared with the transformed model to provide a residual video 
field. The residual distortion between the transformed model and the 
residual video field is resolved to provide updating information for 
10 updating the estimated transformation and to thereby provide a calibration 
correction factor for recalibrating the position of each billboard in the 
store in accordance with the camera sense information. 

The replacement of each billboard is accomplished by use of the 
15 processor 714 (Fig. 7) and the various parameter and billboard stores 
using appropriate software programmes as now described in more detail. 

Figure 8 describes the complete process which allows to determine 
the position of each billboard in the camera's field of view, and render the 
20 corresponding part of the billboard into the frame buffer. Since the 
rendering and later the compositing of the graphics buffer with the video 
buffer by means of chroma-key are known art, we will concentrate on the 
billboard position determination with reference also to Figures 6 and 7. 

25 At the beginning of each video field, the pan, tilt, zoom and focus 

sensors (24,25,26) are read 800. These values, combined with billboard 
, "' data from billboard setup data store 704 and camera data from camera 
intrinsic parameters store 712, enable the detection and recognition of all 
billboards in the camera's FOV, independently of the video signal. The 

30 processing of Fig. 1 consists of a loop on all billboards (m) 802,804. For 
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each billboard, its setup data is retrieved 806 from billboard setup data 
store 704 and used with camera intrinsic parameters 808 to compute the 
perspective transformation 810 from billboard m to current field. The 
replacement billboard information is then stored (812) in a frame buffer. 

5 

Fig. 9 describes the billboard setup data store 900 which consists 
of a separate record 902.. 904 for each billboard in the arena. Such a 
record consists of a static image 906 grabbed in favourable conditions and 
of the corresponding static setup data 908. The record also consist of 
10 dynamic setup data 910 which is computed using the image processing 
means in a process known as dynamic re-calibration which has been 
briefly described above and will be further described with reference to 
Figure 11. An alternative procedure providing static and dynamic 
calibration is described with reference to Figures 20, 21 and 22. 

15 

Fig. 10 describes the setup data (either static or dynamic) 1000 for 
a single billboard. It consists of the sensors 1 readings 1002 at the setup 
instance, the billboard quadrilateral vertex 1004 coordinates and the time- 
code of the setup instance 1006. 

20 

The method of dynamic re-calibration can be explained as follows: 



Due to sensors' drift and inaccuracies, a final calibration table and 
other practical reasons, it is impossible to predict the exact location of all 

25 visible billboards at a given instance. However, at many video fields, a 
billboard's visibility may be such that an exact geometric position 
correction can be performed. Since that position is closer both temporally 
and spatially to the subsequent video fields, it is preferable to rely on that 
"luck shot" by predicting the billboard position relative to its sensors' 

30 readings and exact quad coordinates. Consider for example a billboard 
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which exits the field of view due to camera panning. Having a luck shot 
while it is still highly visible, allows the smooth tracking of the billboard 
by sensors only, when its visibility does not allow any image processing 
means to be applied. 

5 

Fig. 11 depicts the flow diagram 1100 for perspective 
transformation computation. A setup data selection logic 1102 selects 
either the static 1 103 or the dynamic 1 105 setup data from the setup data 
store 806 as described above. This setup data, together with camera 
10 intrinsic parameter is used to compute a sensor-based prediction of the 
perspective transformation 1 104, independendy of the video signal. 

A dynamic re-calibration 1106 based on image processing means 
is then applied to the prediction. It utilizes the video 1 108 and chroma- 
15 key 1110 signals as well as the billboards model image 1112 from the 
setup data store 806 (Fig. 8). Based on a quality factor derived from the 
image processing means, either the sensors-based 1118 or the corrected 
transformation 1 1 16 are output. If the estimated quality of the geometric 
correction is high, then the dynamic setup data is updated 1114. 

20 

Figs. 12,13,14 describe the sensor-based prediction of billboard 
coordinates in the video field. Such a prediction utilizes the sensors 
reading as well as the camera intrinsic parameters. These parameters are 
described in Fig. 12 and have to be tabulated for a dense sampling of the 
25 (zoom, focus) space. The meaning of these parameters is clear from 
Figure 13 to which reference is now made. 

Let tne set of measurements given by the pan, tilt, zoom and focus 
sensors be represented by the vector (P,T,Z,F). The tilt angle is assumed 
to be relative to the horizon. 

30 
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Consider an object point whose image at some setup instance is, at 
frame-buffer coordinates (x s , y s ). Let also the sensor measurement vector 
at that instance be (P s , T s , Z s> Fs). 

5 At another instance, the prediction instance, let the sensor 

measurements vector be (P P , T P , Z P , F p ). It is required to predict the 
location of the object point in frame-buffer coordinates (possibly out of the 
actual frame-buffer), (x P , y P ). 

10 To enable the procedure we define the setup rotation matrix as 

shown at 600 and the prediction rotation matrix is defined as shown at 
602. 

Then, the perspective transformation matrix between the two image- 
15 plane coordinate systems is given as shown at 604 and 1402 (Fig. 14). 

R SP is a 3*3 matrix with row and column indices ranging from 0 to 
2. R SP [i][j] denotes the term in row i, column j in the matrix. Thus, 
given the setup image-plane coordinates of the object point (u s , Vj), the 
20 predicted location of the object point in image-plane coordinates (u P . v P ) 
is given as shown at 606, 1404. 

Image-plane to frame-buffer coordinate transformation is achieved 
as shown at 608, 1406. Aberration compensation is achieved as shown at 
25 608, 1406 (Fig. 14) to provide predicted frame buffer billboard 
coordinates and perspective transformation data. 



An effective way of deriving these parameters for a specific 
(zoom, focus) pair is described in [J. Weng et al., Calibration of stereo 
30 cameras using a non-linear distortion model, IEEE 10th Intl. Conf. Pattern 
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Recognition (1990), pp. 246-253]. The image processing means for 
geometric correction, which allow also the process of re-calibration, is 
now described with reference to Fig. 15. 

5 The image processing means for geometric correction of sensors- 

based prediction are based on the differential method for motion estimation 
[C. Cafforio and F. Roca, The differential method for motion estimation, 
in: T.S. Huang, eg., Image sequence processing and dynamic scene 
analysis, Spring, Berlin, 1983, pp. 104-124]. Let C be the current video 
10 field and let M be the static billboard setup image, perspectively 
transformed according to the sensors-based prediction. Here we consider 
only luminance images. Ideally, M and C are identical within the support 
of the billboard quadrilateral. Actual differences may include: 

• Occlusion present in C but not in M. 

15 • Geometric errors due to sensors and intrinsic camera 

parameters errors. 

• Luminance changes. 

Neglecting for the moment any difference which is not due to 
20 geometric errors, consider a point (x,y) inside the support of the billboard 
quadrilateral. Let (p,q) be the local geometric error then we may write 
for the luminance signals of the respective images: 
M(x + p,y + q) = C(x,y) 
Under the assumption that the error is small, one may write a 
25 Tailor series expansion: 

M{x + p,y + q) = M{x,y) + p — + q — + {Second order terms) 

dx dy 

Neglecting the second order terms and denoting the spatial 
derivatives 
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dM/dx = H 
dM/dy = V 

we obtain 

C(x,y) - M(x,y) = pH + qV 

5 

Also denoting the differences C(x,y) - M(x,y) by D we obtain 
D = pH + qV 

The equation above holds, locally. For a global billboard solution, 
10 and small error assumption we may use the perspective model [G. Adiv, 
Determining Three-Dimensional Motion and Structure from Optical Flow 
Generated by several moving objects, IEEE Trans. Pattern Analysis and 
Machine intelligence, 7, pp. 384 - 401, 1985]. 



15 &,y> - iiivii 

a 1 x + ay + 1 
a A x + ay + a, 
&jx + ay + 1 



20 The coefficients al , . . ,a8 are computed by minimizing the following 

expression: 
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25 



Now, the perspective transformation matrix (based on sensors' 
prediction) is multiplied by: 
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5 The matrix obtained can be considered to be the updated prediction 

of billboard perspective. 

In a practical environment the following considerations may apply. 

• Occlusion may cause major problem in this formulation, 
since if pixels from occluding and moving objects participate 

10 in the minimization of the expression above they might bias 

the solution significantly. Preferably, such pixels are 
discarded from processing by using chroma-key panels. A 
key signal output by a chroma-keyer, is preferably utilized 
to discard these pixels. 

15 • Luminance variations can be minimized by pre-processing 

the current video field, using histogram matching 
techniques. 

• The prediction-correction process may require 2-3 iterations 
to converge. 

20 • Noise immunity and convergence can both be enhanced by 

pre-smoothing the images. 
Thus the billboards 14 etc are in accordance with the present 
invention chroma-key boards and occlusion is by colour discrimination 
using the normal chroma-key techniques. These techniques will enable 

25 perfect occlusion providing that the players do not wear any colour which 
is the same as the board. This may not always be possible and it is 
proposed in accordance with a particular embodiment of the present 
invention to use boards which can rotate or otherwise change to a second 
or third colour. For example, three colours may be blue, green and red 

30 which may be selected when the colour of the players strips are known. 
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Alternatively if it is required to display a billboard in an area of the 
pitch or a surrounding area then such an area must be selected to be of a 
known colour which can then be recorded in the chroma-keyer as a 
chroma-key colour. 

5 

In a preferred embodiment chroma-key apparatus can comprise the 
ULTIMA Tl £-7 digital video image compositing device from 
ULTIMATTE Corp., 20554 Plummer St., Chatsworth, CA 9131 1, USA. 

10 The backing colour can be selected between blue, green and red. 

In order for the chroma-keyer to calculate all parameters necessary to 
perform proper image composidng, the system requires a sample of the 
background colour as reference. This step can be done automatically by 
scanning the image and detecting the purest and brightest colour. 

15 Advanced chroma-keyers enable the user to manually select the area to be 
sampled. 

In a particular embodiment it is proposed to use a patterned 
chroma-key panel. Calibration of the camera sensors can then readily be 
20 accomplished by comparison of the pattern on a pixel by pixel basis. The 
pattern on the billboard panel should preferably have critical dimensions 
less than anticipated sensors error (projected to world coordinates). 

In summary, the above system can operate even in extremely poor 
25 weather conditions since the electronic processing circuitry knows exactly 
where each billboard is situated and does not rely on any analysis of the 
video image to detect the billboard. In the event that the video image is 
so distorted that recalibration cannot be carried out with reasonable 
certainty, then the original camera parameter settings can continue to be 
30 used since the video image as viewed will be of poor quality and thus the 
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viewer will not notice an error of one or two pixels in the positioning of 
the replacement billboard which will require to be displayed in an 
equivalent quality which matches the poor quality video image. 

5 In a further preferred embodiment of the present invention, the 

problem addressed is that of having billboards situated in different 
positions in a stadium as shown in Figure 16. 

In such conditions the lighting of billboards, 1,3 and 5 will be 
10 different because of the location of lights 7, 9 and 13. Also this lighting 
can change all the time during the game. 

If such billboards are chroma-key boards ail of the same colour 
then the billboards will all appear to be slightly different colours due to 
15 the different lighting conditions. 

A fixed adjustment of a global backing colour might result in partial 
object background separation by the chroma-keyer. 

20 In the present invention it is proposed to provide spatial adaptation 

of the backing colour map so that the chroma-keyer can corrccdy 
recognise each billboard. This can be provided by storing in store 704 
(Figure 7) a spatial map providing information relating to the colour of 
each chroma key board. 

25 

Thus, the chroma-keyer will compare the colour in each video 
location with a specific colour associated with the billboard in that 
location. 



30 



In a preferred embodiment the locations of the billboards may be 
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identified by "painting" a slightly enlarged box surrounding the billboard 
to identify the location. Such boxes are identified as 1\ 3' and 5" by 
dotted lines in Figure 16. 

5 The system will track the backing colours over time and therefore 

will continuously update, to ensure correct identification, once correctly 
set up. 



The operation of the system is as follows. 

10 

Firstly with reference to Figure 17, minimum and maximum levels 
are set for U and V. These should be wide enough to encompass all 
billboards which are reasonably lit. 



15 Then for each billboard, as lighting conditions change, an 

adjustment can be made to its stored values, as shown in Figure 18, which 
assumes occlusion of billboard 1 by an object 13. An inner box 1" is 
defined to ensure only pixels from within 1 are considered. Most 
occluding pixels can be discarded as these will be of a different colour. 

20 Then all pixels (Yu V) inside FOV and billboard quad 1 " are measured and 
an addition to the average (which is backing colour average UV over 
billboard) is made if :- 



u nin < u > u 0 

25 or V mt0 < V > V„ 



The inventors have recognised a further problem which arises from 
the use of chroma-key billboards in a stadium. Due to the variable 
lighting as described above, each billboard will appear on the video image 
30 as a slightly different colour. In order to transmit correct occlusion 
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information it is necessary to transmit an occlusion map for each 
billboard. 

In accordance with a preferred embodiment of the present invention 
5 it is proposed to transmit, for each billboard, a perfect background colour 
and to then allow a chroma-keyer in each receiving station to introduce the 
occluded portions by normal chroma-key procedures. 

Consider now the billboard arrangement as shown in Figure 16. 
10 Each billboard 1 ,3 and 5 will, because of its different lighting conditions, 
appear to be a different colour even though this colour may be within the 
maximum and minimum limits as set out in Figure 17. 

In accordance with this preferred embodiment of the present 
15 invention, the transmitting apparatus (see Figure 6) will transmit a perfect 
chroma-key colour within the area of the billboard and will also transmit 
the coordinates of the quadrilateral formed by the billboard. 

In this way the receiving station only has to decode/extract the 
20 quadrilateral coordinates of the billboard and then within that quadrilateral 
replace those pixels which are the perfect chroma-key colour by the 
replacement billboard. Those pixels which are not a perfect chroma-key 
colour are not replaced. 

25 In accordance with this system it is not necessary for the chroma- 

keyer at the remote receiving station to be able to recognise different 
billboards and to have to store different chroma-key values for each 
billboard. Also it is not necessary to transmit any occlusion information 
since occlusion by the chroma-keyer will be relatively simple at each 

30 remote location. 
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With reference to Figure 7, the billboard position and backing 
colour store 704 knows the position of each billboard and a control output 
7042 from the store is used, in combination with the video output 7102 to 
provide inputs for a backing colour processor 7044 which can change the 

5 colour of the billboard within the coordinates provided by store 704. The 
output of processor 7044 is used to control a backing colour store 7046 
which changes the colour of the billboard within the required coordinates 
and can also provide the coordinates to the video output 720 for the 
remote receiver. These may be transmitted by a standard video data 

10 transmission system. 

An exemplary remote receiver is shown in Figure 19. Video data 
is received at receiver buffer 1900 and split and delayed 1902, 1904. 

15 Billboard coordinate store 1906 stores the transmitted billboard 

coordinates and in combination with graphics generator 1908 and 
replacement billboard image store 1910 provides an output signal to a 
combiner/chroma-key er 1912 to produce the desired, occluded billboard 
on the screen. 

20 
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With reference now to Figures 20 to 22 in a further embodiment 
the set up data stored in store 704 is modified prior to any event being 
televised. 

5 The modification comprises the addition of dynamic set up data as 

well as the static image and static set up data shown in Figure 9. 

The additional data may be used instead of the dynamic 
recalibration set up data 910 shown in Figure 9 or could be used in 
10 addition. 

In a preferred embodiment it is assumed that the additional data is 
used instead of the dynamic recalibration procedure and this is now 
described. 

As an introduction, the problems associated with replacing 
billboards with virtual billboards are discussed. The same problem is 
identifying the position, size and perspective of the original billboard and 
then replacing this with the virtual or replacement billboard. 

In a static camera situation there is no real problem once the 
original co-ordinates have been recorded providing that the camera sensors 
do not drift substantially over time. 

However, the inventors have found that during rapid panning or 
tilting of the camera the co-ordinates 908 of the replacement billboard as 
recorded in the store 900 do not coincide with the actual position of the 
billboard in the stadium or venue. This is because the camera sensors 
exhibit a degree of hysteresis. This can be compensated for by the 
30 dynamic recalibration process already described but this may not be 
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practical in some circumstances such as during rapid panning with 
substantial occlusion of the target billboard. 

The hysteresis could possibly be countered by a simple percentage 
5 error built into the movement of the camera but this does not produce very 
good results because it does not take into consideration the camera angle 
with respect to each billboard nor does it take into account the variability 
in the camera parameter sensors with angle. 

10 In the present invention therefore in an alternative embodiment, in 

addition to static billboard set up data the billboard set up data store 900 
stores data for each billboard for each camera at least in relation to left-, 
right pan 2002, right-left pan 2004, uptown tilt 2008 and down-up tilt 
2006. The data is obtained and stored as now herein described with 

15 reference to Figures 20 to 22. 

Figure 20 shows the store 900 modified to provide, in addition to 
the static image data for billboards 1 to M and the static set up data for 
billboards I to M four ftirther sets of data for each billboard 1 to M and 
20 these are multiplied to provide this data for each camera. 

For each camera the position of each billboard is recorded with the 
camera panning from left to right 2002 and for right to left 2004. The 
panning speed may be selected as the normal speed for the event being 
25 televised. Thus for example for horse racing it could be low but for 
motor racing it could be higher. The position of each billboard is then 
recorded with the camera tilting upwards 2006 and then downwards 2008 
across each billboard. 



30 



For each measurement the camera zoom and focus are preferably 
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set at a known level at which the billboard being analysed is in a 
reasonable view at a reasonable size. The zoom and focus could for 
example be the same as that during the acquisition of the static set up data 
for each billboard so that a direct comparison with the static set up data 
5 can be made. In that case only an error correction figure may need to be 
recorded. 

It is preferable not to select too high a panning or tilting speed 
because at very high speeds the billboards will in any case be blurred and 
10 therefore accuracy of replacement will not be an issue. This procedure is 
followed for each billboard for each camera and the data is then used 
during the event to correct the position of each billboard as the camera 
pans or tilts. 

15 It may be seen that each billboard will be viewed at a different 

angle by each camera and also that the output of each of the sensors on 
each camera may vary dependent on the angle through which the camera 
must turn to view the billboard. By recording the static data and data 
relating to panning and tilting in both directions the replacement billboard 

20 will be accurately positioned in the exact position of the original or real 
billboard both for static shots and when the camera is moving. 

Dynamic recalibration during the event as previously described will 
ensure, except during very fast camera movements with large occlusion, 

25 that the replacement billboard is correctly positioned but the use of static 
and dynamic set up data will also ensure this unless the camera sensors 
drift substantially during an event. Thus providing that the camera sensors 
are of a reasonable quality from the point of view of draft they can be of 
a variable quality with respect to accuracy during panning and tilting. By 

30 careful selection of camera sensors extremely accurate sensors are 
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therefore not required since any variation with respect to camera 
movement is compensated for by the storage of dynamic set up data. 

The data is obtained as described with reference to Figures 21 and 
5 22 as follows. 

Once the static data relating to each billboard image and its static 
set up data (906, 908, Figure 9) has been obtained, sequence 2100 is 
started and the camera is panned 2102 by the operator at a desired speed 
10 relative to the normal panning speed. 

The sensors indicate the direction of pan 2106 and dependent on the 
direction the dynamic data is stored in store 2002 or 2004 in steps 2106 
or 2108 by selection of that store. The sequence for both L-R and R-L 
15 stores 2002 and 2004 is similar and will be described for the L-R store but 
using reference numerals for both stores. 

As the camera pans L-R each billboard is identified from data 
stored in store 906 step 21 10; 21 12. The system asks if the billboard has 

20 previously been recoded dynamically step 21 14, 21 16 and if so it returns 
to the start of the sequence and repeats steps 2104-21 10 until it finds a 
billboard that has not been dynamically scanned. Once a new billboard 
has been found the position (co-ordinates) of the billboard during panning 
is recorded and compared with the static billboard parameters previously 

25 stored (908) in step 21 1 8, 2120. Any error is computed (step 2122, 2124) 
and the errors are stored in L-R and R-L pan stores 2126, 2128 for the 
billboard. The system asks if all billboards recorded in store 908 have 
been dynamically scanned both for L-R and R-L (steps 2130, 2132). If 
not the sequence is continued until the last billboard has been dynamically 

30 scanned and then the program is terminated 2134, 2136. 
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A similar program sequence shown in Figure 22 is provided for 
tilting of each camera. Obviously if the cameras are not either allowed to 
tilt or are unlikely to be tilted to any extent then this sequence and the 
recordal of data in stores 2006, 2008 may not be necessary. 

5 

The sequence is started 2200 and each camera in turn is tilted 2202 
and the direction of tilt determined 2204 by the camera sensors. 
Dependent on whether the camera is tilting up or down dynamic set up 
data is stored in stores 2006 or 2008 in steps 2206, 2208. Both sequences 
10 are similar and only the sequence tilting the camera down will be 
described with reference then to both sequences. 

Each billboard is identified 2210, 2212 from the static image data 
and also from the camera parameters especially Where all real billboards 

15 are the same. The program interrogates the billboard data step 2214, 
2216 to see if the billboard has already been interrogated. If it has the 
program restarts but if not the co-ordinate data of the billboard during 
tilting is compared with the static data step 2218, 2220. The error, if any 
is computed step 2122, 2124 and stored in the stores 2006, 2008 (Figure 

20 20) step 2226, 2228. 

The program then interrogates stores 2006, 2008 to see if all 
billboards have been dynamically interrogated for tilt errors in both up 
(step 2230) and down (step 2232) and if so ends the program steps 2234, 
25 2236. If not the program continues by commencing at the start of the 
sequence until all boards have been interrogated. 

Normally camera zoom and focus will not require the same type of 
dynamic set up data to be stored. However, if particular camera 
30 aberrations are known then these may be compensated for by use of 
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similar dynamic set up data. 

The dynamic data stored in stores 2002-2008 may be used instead 
of or in conjunction with the dynamic recalibration data obtained as 
5 described with reference to Figure 9. Usually however the dynamic set 
up data will obviate the need for recalibration during most types of event. 

During use the system knows by reading the camera sensors 
whether the billboard is being viewed in a static manner or is being 
10 panned past L-R or R-L or tilted past UP or DOWN. In such cases the 
position of the billboard is taken from the static data store and then if 
panning or tilting is occurring, the necessary error corrections are applied. 
Once camera movement ceases the static billboard parameters are reverted 
to. 

15 
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THE EMBODIMENTS OF THE INVENTION IN WHICH AN EXCLUSIVE 
PROPERTY OR PRIVILEGE IS CLAIMED ARE DEFINED AS FOLLOWS: 

1 . Apparatus for automatic replacement of a billboard 
5 in a video image, including a video camera for 

viewing the billboard, characterized in that the 
billboard comprises a chroma key surface with a 
chroma key colour and that the apparatus further 
includes a chroma key unit, including means for 

10 storing a reference colour of the chroma key unit, 

means for detecting the chroma key colour of the 
billboard chroma key surface and means for adjusting 
the reference colour of the chroma key unit to 
conform to the chroma key colour of the billboard 

15 chroma key surface, whereby the billboard surface is 

identified for replacing the chroma key surface of 
the billboard by a -virtual advertisement in a video 
image with occlusion of any foreground object. 

20 2. Apparatus as claimed in claim 1 in which the 

apparatus further comprises a billboard map memory 
for recording the positions of a plurality of 
billboards and for recording the chroma key colour 
of each billboard to be replaced, to thereby provide 

25 a chroma key colour map for a stadium. 

3 . Apparatus as claimed in claim 2 in which the 

apparatus further comprises means for automatically 
adjusting the chroma key colour map in accordance 
3 0 with variations in lighting conditions within the 

stadium. 

4 . Apparatus as claimed in any one of claims 1 to 3 in 
which the camera incorporates an automatic camera 
3 5 orientation measurement means including motion 

measurement means operative to measure the Field of 
View (FOV) of the camera relative to a known 
reference position. 
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Apparatus as claimed in claim 4 including image 
processing means for processing video signals 
generated by the camera in which said image 
processing means includes calibration means for 
periodically automatically calibrating the motion 
measurement means . 

Apparatus as claimed in any one of claims 1 to 5 
further including means for transmitting, for each 
billboard to be replaced, a perfect background 
colour key signal. 

Apparatus as claimed in claim 6 further including 
means for transmitting the co-ordinates of each 
billboard to be substituted. 

Apparatus as claimed in claim 7 further including 
receiving apparatus for reception of the billboard 
coordinates and the perfect background chroma key 
colour and including means for providing a combined 
video picture incorporating occlusion in the 
coordinate areas identified by the perfect 
background chroma key colour* 

Apparatus as claimed in claim 2 further comprising a 
dynamic billboard memory for recording any changes 
in measured position of the billboard as the camera 
pans or tilts. 

A method for automatic electronic replacement of 
billboards in a sports stadium, said billboards 
having chroma key surfaces in a video image 
comprising the steps of: 

i. identifying a first specific area corresponding 
to a billboard in the video image, said 
specific area having a defined first chroma key 
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10 



20 



25 



30 



colour of the chroma key surface of a first 
billboard; 

ii. recording the first colour of said first 
specific area in a chroma key colour store; 

iii. identifying a first electronic replacement 
billboard for substitution into said first 
specified area; and 



iv. replacing by chroma key techniques said first 
replacement billboard in said video image said 
replacement including the step of occluding 
those parts of the first specific area which 
15 are occluded in the video image by an occluding 

object . 

11. A method as claimed in claim 10 comprising the steps 
of: 



i. identifying further specific areas in the video 
image, each further specific area comprising a 
further billboard having a related chroma key 
colour; 

ii. recording the colour of all specific areas in a 
chroma key colour store together with 
information relating to the position of each of 
said specific areas; 

iii. identifying replacement billboards for 
substitution into each of said specified areas; 
and 



3 5 iv. replacing the substitute billboards into the 

correct respective locations with occlusion by 
occluding objects. 
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A method as claimed in claim 10 including the steps 
of: 

updating the recorded chroma key colours for 
each specific area at appropriate time 
intervals to compensate for lighting changes 
within a stadium. 

A method as claimed in claim 11 or claim 12 further 
including the steps of: 

i- transmitting together with video image 

information data identifying the specific area 
comprising location and size of each of a 
plurality of billboards; 

ii. transmitting for each specific area identified 
as a billboard a perfect chroma key colour; 

iii. receiving in a receiver the data identifying 
each of plurality of billboards to be replaced 
and the perfect chroma key colour information 
for said specific areas; 

iv. comparison of the perfect chroma key colour 
with a preselected matching perfect chroma key 
colour in a chroma key apparatus in said 
receiver; 

v. - substitution of a plurality of billboards 

selected locally at said receiver into the 
specific area identified for each billboard; 
and 

vi. inserting occluding objects by chroma key 
techniques for each billboard using a chroma 
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keyer in which the chroma key colour is set to 
detect the perfect chroma key colour. 
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1. setup rotation matrix 




cos(P s ) 0 -sin(Ps) 


"l 0 0 " 






0 1 0 


0 cos(Ts) sin(T s ) 






sin(P s ) 0 cos(P s ) 


0-sin(T s ) cos(T s ) 
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2. prediction rotation matrix 



cos(Pp) 0 -sin(Pp) 

0 1 0 
sin(Pp) 0 cos(Pp) 



1 0 0 
0 cos(T p ) sin(T p ) 
0-sin(T p ) cos(T p ) 



602 



3. setup to prediction perspective transformation 

R«p =R S " 1 Rp 



4. setup to prediction point transformation 



Up = 



Rsp|0)|01u 5 + R SD |l|(Q|y s + R SD 12||0| 
Rsp|0|[2|u s + R sp |lj|2)v s + R sp |2||2| 

Rsp [0||11us+RspHHHvs + Rsd 121111 
Rsp |01|2|u s + Rsp |1 ||2|v, + Rsp |2|12] 
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5. image plane to frame-buffer 

x = (u-A u (u,v» Mx+ Xc 

y = (v-A v (u,v)) M Y + V'c 




is. 13 



6. aberrations 

A u (u,v) = (g 1 + g 3 )u 2 + g 4 uv + g lV 2 + u(u 2 + v 2 ) 
A v (u,v) = g 2 u2 + g 3 uv +(g 2 + gjv 2 ^-^ v(u 2 + V 2) 



•610 



SUBSTITUTE SHEET (RULE 25) 



CA 02231376 1998-03-06 



WO 97/09822 



PCT/GB96/02224 



10/16 



(Pp,T p ) 
from sensors 



(Ps < T s ) 
from billboard 
setup — 



image- plane billboard 
quadrilateral vertex 
coordinates (from billboard 
setup data store) 



1 



compute setup to 
prediction coordinate 
transformation matrix 
(eq. 3) 



V 



.1402 



setup image plane to 
frame-buffer 
coordinates 
(inverse of eq. 5,6) 



apply perspective 
transformation 
(eq. 4) 



-1404 



prediction coordinates: 
image-plane to frame 
buffer 
(eq. 5,6) 



camera parameters from 
camera intrinsic 
parameters store 



-1406 



predicted frame-buffer 
billboard coordinates and 
perspective 
transformation 



Fig. 14 



SUBSTITUTE SHEET (RULE 26) 



CA 02231376 1998-03-06 



WO 97/09822 



PCT/GB96/02224 



key 
signal 



video signal 

L_ 



11/16 

sensor-based 
transformation 



smooth 
video field 
image 



n = l 



smoothed - 
field image 



compute 
spatial 
derivatives 



gradient 
image 



geometrically transform 
model static image data 
according to perspective 
transformation estimate 



transformed 
model 



smooth transformed 
model image 



compute geometric 

correction, by 
minimization on all 
background billboard 
pixels 



update perspective 
transformation 
estimate 



Fig- 15 




n 



n + 1 



compute quality 
factor 



output transformation 
and quality factor 



SUBSTITUTE SHEET (RULE 26) 



CA 02231376 1998-03-06 



WO 97/09822 



PCT/GB96/02224 



12/16 




Fig. 16 



255 



0,0 



Vmax Umax 



Umin Vmin 



255,255 



Fig. 17 



U 
255 




Fig. 18 



SUBSTITUTE SHEET (RULE 26} 



CA 02231376 1998-03-06 



WO 97/09822 PCT/GB96/02224 

13/16 






0* 


> 


□ 


u 














CO 



a" 

S z w 

O ~ OS 

J H 
J O K 

5 S 



CD 
O 
CD 



Os 



SUBSTITUTE SHEET (RULE 26) 



CA 02231376 1998-03-06 



WO 97/09822 PCT/G B96/02224 

14/16 



BILLBOARD STATIC AND DYNAMIC 
SET I'P DATA STORE 


DYNAMIC CALIBRATION DATA 


TILT 
DOWN 


B/B 1 


2008' 


B/B M 


TILT UP 


B/B 1 


2006 

1 i 


B/B IV! 


z 5? 


B/B 1 


2004 


B/B M 


Z OS 


B/B 1 


_o 


B/B M 


STATIC SET 
IIP DATA 

1 
1 


B/BOARD I 


ae 
O 


B/BOARD M 


STATIC 
IMAGE 
DATA 


B/BOARD 1 


© 


B/BOARD M 



SUBSTITUTE SHEET (RULE 26) 



CA 02231376 1998-03-06 



WO 97/09822 



PCT/GB96/02224 



15/16 



START 



-2100 



PAN CAMERA 



-2102 



^2104 



IS 

CAMERA\ N 
PAN LEFT TO. 
RIGHT r 

^2106 



SELECT L-R 
PAN STORE 



21082v 



2110 



2112 



IDENTIFY B/BOARD 
FROM STATIC IMAGE 



2l 



SELECT R-L 
PAN STORE 

I 



IDENTIFY B/BOARD 
FROM STATIC IMAGE 



Y ^B/BOARD* 
DETAILS ALREADY" 
iCORDED ?_ 

N 



-2114 



2116 



2118 



COMPARE POSITION OF 
B/BOARD WITH STATIC 
B/BOARD PARAMETERS 




COMPARE POSITION OF 
B/BOARD WITH STATIC 
B/BOARD PARAMETERS 



COMPUTE ERROR IN 
B/BOARD POSITION 

I 



2122 



2124- 



STORE ERROR OF 
IDENTIFIED B/BOARD 
IN L-R PAN STORE 



2126 



2128- 



COMPUTE ERROR IN 
B/BOARD POSITION 

I 



STORE ERROR OF 
IDENTIFIED B/BOARD 
IN R-L PAN STORE 



HAVE" 
"LAST B/BOARD" 
DETAILS BEEN 
N\ RECORDED 
= M> 

. Y 

2134. 



END 



-2130 



2132- 



Fig. 21 



HAVE" 
"LAST B/BOARI 
DETAILS BEEN 
RECORDED 

= M 

Y S 



END! 



-2136 



SUBSTITUTE SHEET (RULE 26) 



CA 02231376 1998-03-06 



WO 97/09822 



PCT/GB96/02224 



16/16 



START 



-2200 



TILT CAMER"aT -^_ 2 202 




2208 



SELECT UP- 
DOWN STORE 



2210 



2212 



IDENTIFY B/BOAKD 
FROM STATIC IMAGE 



1 



SELECT DOWN 
- UP STOR E 

r 



IDENTIFY B/BOARD 
FROM STATIC IMAGE 



Y -^B/BOARD 
DETAILS ALREADY 
RECORDED ? 

N 



2214 



2218 



2216 



COMPARE POSITION OF 
B/BOARD WITH STATIC 
B/BOARD PARAMETERS 

1 




2220 



COMPARE POSITION OF 
B/BOARI) WITH STATIC 
B/BOARD PARAMETERS 



COMPUTE ERROR IN 
B/BOARD POSITION 



I 



•2222 



2224 



COMPUTE ERROR IN 
B/BOARD POSITION 



STORE ERROR OF 
IDENTIFIED B/BOARD 
IN U-D TILT STORE 



-2226 



I 



STORE ERROR OF 
2228H IDENTIFIED B/BOARD 
IN DOWN-UP TILT STORE 



HAVE 
LAST B/BOARir 
DETAILS BEEN 
RECORDED 
in = M 



-2230 



2232- 



2234 — I END 



Fig. 22 



iiAvir 

LAS I B/BOARD 
DETAILS BEEN 
RECORDED 

sin = M •> 



Y 



"END] 2236 



SUBSTITUTE SHEET (RULE 26) 



CO 

o 

< 



